Surrounding intelligent motion sensor

ABSTRACT

A wearable proximity warning device called SIMS (Surrounding Intelligent Motion Sensor) is provided that uses a novel method of processing images from a high frame rate digital camera to detect human threats from behind and determine if there are any approaching threats by using novel pixel counting and threat detection analysis algorithms. The device is worn on the back of the body either by use of a belt clip or with chest straps. The user may use select from a variety of warning options from the device including audible warning tones, device vibration or smartphone SMS/MMS text messaging. Stored video is saved by the SIMS device and may be periodically uploaded to secure cloud storage. The device contains a rechargeable battery that may be recharged using a USB port.

CROSS REFERENCES TO RELATED APPLICATIONS

Not Applicable

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable

THE NAMES OF THE PARTIES TO A JOINT RESEARCH AGREEMENT

Not Applicable

INCORPORATION BY REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC OR ASA TEXT FILE VIA THE OFFICE ELECTRONIC FILING SYSTEM

Not Applicable

STATEMENT REGARDING PRIOR DISCLOSURES BY THE INVENTOR OR A JOINTINVENTOR

Not Applicable

BACKGROUND OF THE INVENTION

The present invention relates to the field of proximity detectiondevices, and more specifically to a wearable proximity detection deviceand a novel method of providing situational awareness and warning to theuser of approaching threats from behind the user. There are currentlyover 500,000 correctional officers in the US with over 13,000 assaultsper year. Unfortunately, there are currently no products available tohelp corrections officer professionals reduce their risk of beingassaulted from inmates who are approaching them from behind. Besidescorrectional officers there are many other user groups who could benefitfrom a device to warn of a threat approaching from behind such as policeofficers, military personnel, and even civilian users such as joggersand runners for example.

There are several proximity detection devices discussed in the priorart. As a first example, U.S. Pat. Appl. Pub. No. 2015/0172545 publishedin June 2015 by Szabo et al entitled “Situational Awareness byCompressed Display of Panoramic Views” discloses a device that stitchesvideo images from a camera to provide a 360 panoramic view of thesurroundings to provide situational awareness. The Szabo device is ahead mounted device intended for military and firefighters to wear onduty. Although the Szabo device could be used effectively for militaryand firefighting applications, it would be much too expensive for theaverage civilian to afford due to the complex array of imaging sensorsand image processing hardware required.

A second example is U.S. Pat. Appl. Pub. No, 2017/0263107 published inSeptember 2017 by Doyle et al entitled “Approaching Proximity WarningSystem, Apparatus and Method” discloses a wearable device that alertsthe user of threats approaching from behind. The Doyle device uses shortrange radar to transmit and receive electromagnetic waves that bounceoff of objects and processes the radar information using an algorithm todetermine if there is an approaching threat from behind. The Doyledevice is meant to be worn on the back of the user and will warn theuser by various means such as by an audible signal or a mechanicalsignal such as by vibration of the device.

There are several inherent design flaws in the Doyle device that areaddressed in the present invention. The first flaw inherent design flawin the Doyle device is that it uses short range radar technology whichhas typically a maximum effective range of only 100 meters (328 feet).The patent pending device of Doyle and Cleghorn is disclosed on theinternet by a New Mexico company known as DefendSix (www.defendsix.com).The DefendSix website product specification lists the detection range asapproximately 25 meters (82 feet). With such as short detection range,there is a very high risk that the user of a DefendSix device may nothave enough warning time to take evasive action if a threat such as arunning person is approaching from behind. For example, based on atypical human running speed of 15 mph (22 feet per second), the Doyledevice would first alert a static user when the person was 82 feetbehind him. This would give the user only 3.7 seconds to take evasiveaction before the person contacts him if the user was standing still.Although a running user would have a few more seconds based on theirspeed differential, the point to be made is that the very limited rangeof the Doyle device leaves a very short time to take evasive actionbefore the threat has reached the user.

The second inherent design flaw in the Doyle device is that because ituses radar it cannot discriminate a real threat from a false positivethreat. For example, a bird flying towards the user will generate awarning to the user just like a person with a knife running from behindwould. Although the radar would likely always warn the user once the 82foot detection range is reached, the user over time would likely receiveso many false positive warnings that he would start becomingdisappointed with its effectiveness and basic lack of intelligence ofthe device.

Yet another inherent design flaw in the Doyle device is that itsprobability of detecting a target is based on a property of an objectknown as its radar cross section. The radar cross section of an objectdepends on many variables such as the object's size, surface area, shapeand material. While some objects such as a large jogger running straightat a person from behind may have a large enough radar cross section forthe device to warn the user, a small object such as a thrown baseball ora bicyclist might have too small of a radar cross section to reach thedetection threshold depending on its distance from the radar. Becausethe Doyle device uses low power short range radar, the variable natureof radar cross sections will likely result in either some threats notbeing picked up the device or being picked up approaching too close tothe user to safely take evasive action. Indeed, there is a greatunfilled need to supply corrections officers, police officers, militarypersonnel and security concerned civilians with a proximity warningdevice that can reliably pick up targets several hundred feet away.

BRIEF SUMMARY OF THE INVENTION

The present invention provides a wearable proximity warning device andnovel threat detection method that resolves the aforementioned problemsin the prior art by using a high frame rate digital camera with advancedimage processing algorithms to detect threats from several hundred feetaway and track them in real time and immediately warn the user if anythreats are approaching him from behind. The present invention shall bereferred to throughout the remainder of this specification asSIMS—Surrounding Intelligent Motion Sensor.

It is a first object of SIMS to provide a compact, lightweight proximitywarning device that can detect threats approaching from behind at amaximum distance of 500 feet. This is well beyond the typical range ofnon-optical methods such as short range radar (max. detection range is−330 feet).

It is a second object of SIMS to provide a threat detection capabilitythat uses human recognition software to identify potential threats andimage processing software to determine if any potential threats areapproaching from behind.

It is a third object of SIMS to provide the user with multiple warningoptions for when an approaching threat has been detected includingaudible tones, vibration and Bluetooth based secure cellular phone SMStext messages or video snapshots.

It is a fourth object of SIMS to store recorded video on the cloud forsecure data storage.

It is a fifth object of SIMS to allow the device to be attached to thebody either by using a chest belt or by a belt clip.

It is a sixth object of SIMS to provide the wearable device with a USBrechargeable battery that provides at least six hours of continuoususage.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a front view of the SIMS device.

FIG. 2 is a rear view of the SIMS device.

FIG. 3 shows a female user wearing the SIMS device using the chest beltoption.

FIG. 4 shows a male user wearing the SIMS device using the belt clipoption.

FIG. 5 is a component diagram of the SIMS device.

FIG. 6 is a functional block diagram showing the functionalrelationships of the various components of the SIMS device.

FIG. 7 is a flowchart showing how an approaching threat determination ismade in the SIMS device.

FIG. 8 is a flowchart showing the various warning methods that the SIMSdevice will use if an approaching threat from behind has been detected.

FIGS. 9(a)-9(c) show a sequence of time elapsed images of runningjoggers to demonstrate the capability of SIMS to detect humans using ahuman recognition detection algorithm.

FIGS. 10(a)-10(c) show the capability of a human to pixel countconversion algorithm to count pixels of the detected humans fromprevious FIGS. 9(a)-9(c).

FIG. 11 is a table showing the results of the threat detection algorithmto detect an approaching threat by an increasing pixel count of detectedhuman T1.

DETAILED DESCRIPTION OF THE INVENTION

Referring now to the drawings and in particular FIG. 1, the presentinvention is designated by reference numeral 1000 and shall be referredto hereafter as the SIMS device. The SIMS device has a front housing 10that is preferably made from a strong plastic such as ABS or nylon. Adigital camera lens 70 is located in the center of the housing and hasthe typical features of modern digital cameras including a focusingcapability and an adjustable aperture to adjust the brightness of theimage. A USB port 97 is located at the bottom end of the SIMS device andprovides a means for the recharging of an internal rechargeable battery99 (not shown).

Referring again to FIG. 1, there are two belt or strap attachment points55, one on each side of front housing 10. These attachment points allowfor a belt or strap to be secured to them as one method of wearing theSIMS device on the body. A power on/off button 40 is shown located atthe upper left hand corner of the SIMS device and provides a means toturn SIMS on or off. A Bluetooth activation button 20 is located belowthe power button 40 and when this button is pressed it will enable theSIMS device to use Bluetooth cellular phone warning options that shallbe explained later in the specification. The SIMS device can provideeither an audible tone warning or a vibration warning depending on whichposition the warning mode selector switch 30 is in. For example, in FIG.1, the mode selector switch 30 is shown in the audible tone warningposition. When the audible tone warning mode is selected, an internalspeaker 85 (not shown) will produce sound that will be emanated fromspeaker holes 87 shown in the top center section of the SIMS device.

Referring next to FIG. 2, the back side of the SIMS device 1000 isshown. A belt clip 50 is shown centered on the rear housing 15 of theSIMS device. The belt clip 50 provides an additional means of wearingthe SIMS device on the body.

The two methods of wearing the SIMS device 1000 on the body are shown inFIGS. 3 & 4. FIG. 3 shows a woman wearing the SIMS device using a cheststrap that wraps around her chest and is secured to the SIMS deviceusing the body strap attachment points 55 shown previously in FIGS. 1 &2. While the user wears the SIMS device and when it is powered on, itwill provide continuous monitoring for any threats approaching the userfrom behind. The SIMS device 1000 may also be worn on the body by usingthe belt clip 50 located on the backside of the SIMS device to secure toa belt as demonstrated by the man wearing the SIMS device as shown inFIG. 4. It is recommended to only wear the SIMS device at waist heightor higher in order to maximize the field of view that the camera willhave when in use.

Referring next to FIGS. 5 & 6, the functional relationships of thevarious components that make the SIMS device 1000 shall next beexplained. There are 16 essential components that comprise the SIMSdevice. These components are all shown in FIGS. 5 & 6 and are numberedfor reference as follows:

-   -   Motherboard —96    -   Camera Control Board —65    -   Camera Lens & Variable Aperture —70    -   Image Sensor —72    -   Image Processing Board —75    -   Power On Button —40    -   Rechargeable Battery —99    -   Battery Recharging Board —98    -   USB Port —97    -   Audio Board —95    -   Speaker —85    -   Vibration Device —90    -   Smartphone Communications Board —80    -   Antenna —81    -   GPS Receiver —82    -   Audio/Vibrate Warning Mode Selector Switch —30    -   Bluetooth Activation Button —20

The motherboard 96 is the central control board that manages all of theoperations of the SIMS device. The motherboard receives input from theaudio/vibrate mode switch 30 and based on the input received will eitherwarn the user of an approaching threat by either activating the audioboard 95 or the vibration device 90. The motherboard is powered by arechargeable battery 99 and also regulates the DC power inputs to all ofthe separate modules under its control. When the power on button 40 ispressed it allows DC power to flow from the rechargeable battery to themotherboard. When the Bluetooth activation button 20 is pressed it sendsa signal to the motherboard to activate the smartphone communicationsboard 80. Once the SIMS device is turned on, the motherboard activatesthe camera control board 65. A GPS receiver 82 continuously sends thedevice's location (latitude and longitude coordinates) to themotherboard while the SIMS device is operating. This GPS location datacan later be used as part of the text of an SMS text detected threatwarning message or included as part of an MMS text message that includesa picture showing the detected threat. In the case of the MMS textmessage, the GPS location data could be embedded as part of the picturesent so linked smartphone users can access the location of the SIMS userfor assistance.

The camera control board 65 controls the operation of the auto focuscamera with variable aperture 70. Based on the lighting conditionsduring use, the camera control board will regulate the aperture size toallow more or less light into the camera in order to obtain a sufficientcontrast to process the images. Images are taken at a specified framerate (images per second) and are focused onto the image sensor 72 whichis preferably a charge-coupled device (CCD). The image sensor iscomposed of thousands or pixels (picture elements). The light from eachpicture image captured is converted to a digital signal representationby the image sensor and sent to the image processing board 75.

The image processing board 75 is the component of the SIMS device thatperforms the highly complex tasks of threat detection and monitoring.First, a human recognition detection algorithm is used to identifyhumans from the images. Once a human is identified, the humanrecognition detection (HRD) algorithm 500 will continuously track theidentified human until it leaves the field of view of the camera. Whilehumans are being tracked by HRD 500, a separate human to pixel count(HPC) image conversion algorithm 600 is used to count the total numberof pixels that each detected human represents in each image processed.The pixel counts vary over time based on how far away the detectedhumans are from the SIMS device. Humans approaching the SIMS device willhave increasing pixel counts over time and humans that move further awayfrom the SIMS device will have decreasing pixel counts over time. Thehuman pixel counts from the image processing board 75 are continuouslytabulated and sent to the motherboard 96. These human pixel counts arethen supplied as inputs to a threat determination (TD) algorithm 700that is embedded firmware and is used by the motherboard 96. This novelmethod of detection of an approaching threat shall be explained indetail later in the specification.

The smartphone communications board 80 provides the capability ofsending a warning message of a detected approaching threat tosmartphones that have the SIMS device app installed. The smartphone usersimply enables the SIMS device app by going into the smartphone settingsand enabling the app. Once enabled, the smartphone app will useBluetooth to connect the SIMS device to the user's smartphone.

The smartphone communications board 80 performs three independentfunctions. The first function is to send a Multimedia Messaging Service(MMS) picture and text message 200 showing the first image frame that isconsidered as a detected approaching threat by SIMS plus a short warningmessage. An example of a MMS text message 200 would be a text messagesuch as “WARNING: SIMS DEVICE HAS DETECTED APPROACHING THREAT” followedby a picture showing the actual detected approaching threats. The secondfunction is to send a Short Message Service (SMS) text message 300 whichis only a text warning message without any pictures. A final function isthe capability of sending stored video images from the memory of themotherboard 96 to the cloud. The SIMS user would enable these threefunctions as options that would displayed in the SIMS device app.Hardware to support the smartphone communications board includes an RFantenna 81 to transmit the data wirelessly to the selected SIMS enabledsmartphones and also optionally to the cloud for secure video storage.

Another option in the SIMS device app is to allow the user to select aspecific audio warning tone to use when the audio warning mode isenabled. The selected warning tone is then stored as an audio file inthe motherboard and sent to the audio board 95 whenever a threat isdetected. The audio board then sends the tone signal to the speaker 85to play the sound. Yet another option in the SIMS device app is to allowthe user to select the duration time and intensity of the vibrationwarning to use when the vibrate mode is enabled. The selected vibrationtime and intensity is then stored as a file in the motherboard's memoryand is used to produce the desired vibration warning via the vibrationdevice 90.

The USB port 97 is used for the recharging of the SIMS device wheneverthe rechargeable battery 99 level is low. The USB port interfaces with abattery recharging board 98 which is essentially a step down transformerthat converts the DC voltage from the USB power to a lower voltage tocharge the battery. Once the battery 99 has been fully charged, thebattery recharging board 98 will switch off automatically to preventovercharging.

Referring next to the flowchart shown in FIG. 7, the novel method ofdetection of an approaching threat using the SIMS device 1000 shall beexplained in further detail. The SIMS device uses a novel method ofthreat detection that is far superior to prior art methods due to itsability to first recognize humans from digital camera images. First ahuman recognition detection (HRD) algorithm 500 scans the camera imagefile and identifies any humans. The details of how the HRD algorithmworks are beyond the scope of this specification but would be known toone skilled in the art of artificial intelligence software codedevelopment. Next, a second algorithm defines a boundary around eachdetected human in the image and calculates the number of pixels insideeach detected human boundary. This second algorithm is defined in thisspecification as a human to pixel count conversion (HPC) algorithm 600.The HPC algorithm 600 then sends the pixel count outputs of eachdetected human as inputs to a third threat detection (TD) algorithm 700that is part of the embedded firmware in the motherboard 96. This TDalgorithm 700 is continuously fed pixel counts for each detected humanas each image is processed by the image processing board 75.

Referring again to FIG. 7, the TD algorithm 700 is always run wheneverhumans are detected by the SIMS device using the HRD algorithm 500.Pixel counts at a first time (T1) are then compared to pixel counts at asecond later time (T2) and stored frame by frame in the motherboard'smemory. The elapsed time (T2−T1) between image frames is determined bythe frame rate of the camera 70. The SIMS device would preferably use ahigh frame rate camera (for example 1000 frames per second or 1000 fps)in order to achieve a fast detection time of an approaching threat. TheTD algorithm then compares the pixel counts for each detected human overa short time interval (say 1 to 3 seconds for an outdoor exercisescenario) and determines if the pixel counts are increasing ordecreasing. An increasing pixel count for any detected human isconsidered as a detected threat. If the pixel counts are all decreasingthen the TD algorithm continues to monitor the next sequence of pixelcounts received from the image processing board and the processcontinues this way in a continuous monitoring mode until the SIMS deviceis turned off.

Referring next to FIG. 8, once a threat is detected, the SIMS devicewill take immediate action to warn the user. There are four possiblemethods of warning that can be used by SIMS. The first method is audioalert which plays a user selected audio warning tone from the speaker 85driven by an audio board 95. The second method is vibration alert whichvibrates the SIMS device using an internal vibration device 90 based ona user selected vibration file stored in the motherboard's memory.Enabling the Bluetooth mode on the SIMS device allows two additionalalert methods that send either a Short Message Service (SMS) standardtext only warning text message 300 or a Multimedia Messaging Service(MMS) combined picture and text message that displays the same warningtext message as SMS but adds a picture taken from the digital camera atthe moment of first threat detection.

Referring finally to FIGS. 9 thru 11, an example scenario using the SIMSdevice to detect and count the pixel sizes of potential threats isshown. FIGS. 9(a), 9(b) and 9(c) shows three separate images taken atthree second intervals of three joggers that are running behind a userwearing an activated SIMS device. Initially, all three joggers arerunning together (FIG. 9(a)) but by the final image (FIG. 9(c)) theyhave all been separated from each other. The HRD algorithm 500 hasdetected three humans in the images and for reference these detectedhumans are referred to as T1, T2 and T3. Next, FIGS. 10(a), 10(b) and10(c) show the same time elapsed images that show the detected humanimage areas of T1, T2 and T3 that will be converted to pixel counts.Finally, the results of the HPC algorithm 600 are shown in FIG. 11. Notethat the pixel counts only show one detected human (T1) to be anapproaching threat as only the pixel counts of the detected human T1were shown to be increasing over time. Based on this result the SIMSdevice would have alerted the user of an approaching threat.

The SIMS device 1000 provides a robust threat detection capability usinga series of two image processing algorithms and a threat detectionalgorithm to provide a reliable and fast detection time as compared withprior art examples cited. Although the example shown was for a daytimescenario, other embodiments of the SIMS device would allow the SIMSdevice to be used for nighttime military patrol applications. In suchnighttime applications for a military version of the SIMS device, themain difference in hardware would be the inclusion of a second infraredcamera that could be deployed for nighttime approaching threatdetection.

What is claimed is:
 1. A wearable device for detection and warning ofapproaching threats from behind comprising: a housing consisting of afront and a rear housing that mate together to form the exterior of saiddevice; a clip mechanism rigidly attached to said rear housing forproviding a means of attachment to an article of clothing worn on ahuman body; slotted apertures located on the left and right sides ofsaid housing for providing a means of attachment to the human body usinga chest strap; slotted apertures located on said front housing forproviding a means of sound projection from an internal speaker; amotherboard that uses a threat detection algorithm to detect anapproaching human threat and controls the activities of separate boardsincluding an audio board, a vibration device, a smartphonecommunications board and a camera control board; said threat detectionalgorithm comprising: a human recognition detection algorithm thatidentifies humans from captured images; a human to pixel countconversion algorithm that converts the total number of pixels fromhumans identified from said human recognition algorithm and representssaid identified humans as a number of pixels; and a threat determinationalgorithm that determines a threat condition based on an increase innumber of pixels from said detected humans identified from said humanrecognition detection algorithm; wherein the audio board connects to theinternal speaker and is configured to provides an audio warning tone tosaid internal speaker when activated by said motherboard; wherein thevibration device is configured to vibrate when activated by saidmotherboard; a switch at the front face of said device for selection ofeither an audio based warning or a vibration warning for warning theuser of an approaching threat; a button at the front of said device foractivation of a Bluetooth wireless smartphone communication link; abutton for turning power on or off to said device; a rechargeablebattery for supplying power to the internal electronic components ofsaid device; a battery recharging board for periodic recharging of saidrechargeable battery; a USB port for providing a means to supply DCpower to said battery recharging board; a smartphone communicationsboard to send approaching threat warning messages to at least one linkedsmartphone by wireless means using either SMS or MMS text messaging; anantenna connected to said smartphone communications board for wirelesstransmission of SMS or MMS text messages to at least one linkedsmartphone; a camera control board for providing focusing and aperturecontrol of a digital camera lens; a digital camera adjustable focus lensand variable aperture controlled by said camera control board; an imagesensor using CCD (charge coupled device) technology to receive focusedimages from said digital camera lens; a GPS receiver to send GPSlocation data of said device to said motherboard; and an imageprocessing board that receives digital images from said image sensor. 2.The wearable device according to claim 1, wherein said digital cameralens has the capability to focus on objects up to 500 feet away.
 3. Thewearable device according to claim 1, wherein said CCD image sensor hasa frame rate between 100 and 1000 frames per second.
 4. The wearabledevice according to claim 1, wherein said device includes a secondcamera for night vision threat detection that operates in the nearinfrared wavelength range of 0.75-1.4 μm (micrometers).
 5. The wearabledevice according to claim 1, wherein said device includes a secondcamera for thermal vision threat detection that operates in the farinfrared wavelength range of 8-15 μm (micrometers).